Hexagon Op queue & dispatch optimizations #16820
Merged
+136
−362
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We're not going to release the buffers without flushing the session queue. So there is no need to inc/dec the refcounts for every request. We also don't need to include those bufs in the response.
We can use more CPU cores now that the dedicated dspqueue polling threads are not used (ie no contention). Also enable more agressive polling for now since we still map Flash Attention (and a few other kernels) to the CPU and those dspqueue threads were keeping the CPU cores are higher clock freqs.
|
@l3utterfly this would be interesting for your use-case (ie APK with |
|
@max-krasnyansky Thank you! I will test this out! |
github-actions
bot
added
script
|
Some server tests are failing, but should be unrelated. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Optimize dspqueue (used for sending Op requests to the NPU) by doing all response processing in-place.
This removes the need for the dedicated read threads (started internally by the dspqueue library) and essentially
eliminates all polling in that path.
(For the curious, the dspqueue CPU side sources can be found here
https://github.com/qualcomm/fastrpc/tree/main/src/dspqueue)
We can also bump the CPU backend thread counts now for the default use-cases since we still rely on the
CPU for Flash Attention and a few other Ops.